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Modeling  and  simulation  technology  is  the  use  of  models  to  develop  data  as  a 
basis  for  making  managerial  or  technical  decisions.  It  can  be  a  valuable  tool  for 
program  managers — but  it  is  one  that  is  vastly  under-used.  This  article  provides 
a  business-case  framework  (a  methodology  to  evaluate  investment  opportu¬ 
nities)  for  program  managers  within  the  Department  of  Defense  to  use  when 
determining  how  to  apply  modeling  and  simulation  in  project  management. 


The  use  of  modeling  and  simulation 
(M&S)  is  widely  misunderstood 
within  the  Department  of  Defense 
(DoD).  M&S  is  the  use  of  models,  either 
statically  or  over  time,  to  develop  data  as 
a  basis  for  making  managerial  or  techni¬ 
cal  decisions  (DoD,  1997).  Models  are 
physical,  mathematical,  or  logical  repre¬ 
sentations  of  a  system,  entity,  phenom¬ 
enon,  or  process.  Simulations  are  meth¬ 
ods  for  implementing  models  over  time. 
Normally,  we  associate  simulations  with 
a  software  program  that  implements  mod¬ 
els  over  time,  within  the  context  of  a  given 
scenario  (Defense  Modeling  and  Simula¬ 
tion  Office,  1996).  Simulations  permit  the 
user  to  assess  variables  and  the  predict¬ 
ability  of  a  single  or  series  of  outcomes. 


Nowhere  is  the  misunderstanding  more 
painfully  obvious  than  within  the  program 
management  offices  of  the  DoD.  Some 
program  managers  believe  M&S  is  para¬ 
mount  to  effective  project  development 
and  place  the  requisite  investment  in  it 
(and  this  article  highlights  examples  of 
some  such  programs).  But  many  program 
managers  remain  both  skeptical  and  sus¬ 
picious.  Recent  government  direction  to 
use  simulation-based  acquisition  in  DoD 
programs  is  an  example  of  a  policy  with 
good  intentions  but  poorly  shaped  execu¬ 
tion.  This  edict  has  been  met  with,  at  best, 
marginal  acceptance,  and  at  worst,  abject 
resentment. 

Such  resentment  and  apprehension 
spring  from  institutionalized  biases. 
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including  DoD  funding  procedures,  that 
work  against  optimizing  the  potential 
gains  of  employing  modeling  and  simu¬ 
lation.  By  far,  the  severest  criticisms  tar¬ 
geted  at  M&S  center  on  the  debate  over 
return  on  investment  (ROI).  As  we  will 
discuss  later,  ROI  is  just  one  of  many  tech¬ 
niques  in  evaluating  the  use  of  M&S  in  a 
program.  Ostensibly,  DoD  has  accepted 
this  new  technology  as  a  means  of  reduc¬ 
ing  costs,  increasing  cost  avoidance,  and 
banking  the  residual  benefits  for  other 
projects. 

Many  program  managers  argue  that  the 
entire  acquisition  system  is  focused  on 
getting  a  project  into  production,  through 
performance  trials,  and  permanently  into 
the  military’s  inventories.  Seldom  are  they 

given  suffi¬ 
cient  funds, 
staff,  or  time  to 
investigate  the 
potential  ben¬ 
efits  of  tools  or 
technologies 
such  as  M&S. 
Importantly, 
leadership  pro¬ 
vides  little  in¬ 
centive  to  capture  data,  build  expensive 
models,  or  conduct  additional  analyses  to 
transfer  M&S  results  to  other  projects. 
Simply  put,  program  managers  are  under 
intense  pressure  to  complete  their  pro¬ 
grams  on  or  under  budget  and  within  time¬ 
lines.  Existing  programs  lack  enticement 
to  develop  new  models  or  simulation  tools 
that  may  have  wider  application  to  other 
programs,  or  that  will  be  much  cheaper  to 
operate  and  sustain.  With  few  exceptions, 
these  occurrences  were  more  a  result  of 
coincidence  than  deliberateness. 


Finally,  perhaps  the  greatest  impedi¬ 
ment  to  M&S  acceptance  is  lack  of  knowl¬ 
edge.  Many  people  do  not  understand  the 
potential  benefits  of  this  technology,  or 
how  to  define  needs  and  produce  the  right 
tools  that  will  help  the  project.  Unfortu¬ 
nately,  this  reticence  may  be  reinforced 
by  an  institution  which  neither  favors  nor 
rewards  risk  takers.  Sometimes,  a  program 
manager  may  not  know  for  certain  if  a 
major  investment  in  M&S  is  warranted. 
If  a  program  manager  wants  to  invest  in 
the  technology  “just  to  see  if  there  is  a 
benefit,”  he  or  she  faces  criticism  if  the 
results  are  not  positive.  Thus,  a  program 
manager  must  weigh  the  costs  of  the  risk, 
and  consequently  few  take  the  chance, 
preferring  more  traditional  approaches — 
the  building  of  expensive  mock  ups;  the 
use  of  labor-,  time-,  and  money-intensive 
trials,  and  incurred  costs  of  waste. 

The  deputy  project  manager  (DPM)  of 
the  Joint  Strike  Fighter  (JSF)  praised  the 
future  of  M&S  while  lamenting  the  mili¬ 
tary’s  reticence  to  embrace  it.  “The  Joint 
Strike  Fighter  Project  achieved  immeasur¬ 
able  benefits  from  its  innovative  use  of 
modeling  and  simulation.  More  important, 
I  was  impressed  with  the  tremendous  in¬ 
tellect  and  drive  of  many  of  the  staff  who 
were  willing  to  try  new  models.  They  did 
simulation  runs — sometimes  up  to  1,000, 
in  order  to  bring  this  project  along  a  de¬ 
velopment  path  to  bring  the  necessary 
technology-enhanced  fighter  into  the  21st 
century.  They  took  the  risks  and  it’s  pay¬ 
ing  off’  ( 1999). 1  The  DPM  made  this  blunt 
statement — “No  high-tech  project  will 
achieve  any  significant  success  if  it  does 
not  incorporate  M&S.  However,  we  in 
DoD  just  aren’t  ready  to  capitalize  on  this 
technology.” 


"Simply  put, 
program  managers 
are  under  intense 
pressure  to 
complete  their 
programs  on  or 
under  budget  and 
within  timelines." 
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J  oint  Strike  Fighter 


While  all  of  this  bodes  well  for  an 
anecdotal  argument  supporting  M&S, 
what’s  needed  is  a  more  reasoned  and 
defensible  process  for  determining  M&S 
investments.  While  few  refute  the  intui¬ 
tive  benefits  of  M&S,  program  managers 
quite  rightly  argue  that  any  tool  must  be 
first  measured  against  its  potential  ben¬ 
efits  before  it  is  used.  For  example,  given 
that  Boeing  spent  $2  billion  on  M&S  for 
its  777  airplane,  are  comparable  levels  of 
investment  affordable  to  program  manag¬ 
ers  of  programs  of  similar  magnitude? 
Obviously,  the  question  is  not  easily  an¬ 
swered.  It  depends  on  a  variety  of  factors, 
including  the  project’s  funding,  period  to 
recoup  the  investment,  and  perceived  ben¬ 
efits  to  developing  and  using  M&S  in  the 
program.  The  crux  of  the  problem  is  the 
dilemma  of  costs  versus  risks  and  the 
potential  return.  Program  managers  need 


a  methodology  to  evaluate  investment 
opportunities. 

Methodology  and  Approach 


The  authors  reviewed  selected  M&S 
efforts  within  civilian  industry  and  DoD. 
We  visited  several  key  manufacturing  and 
service  industries,  which  were  also  wres¬ 
tling  with  the  same  subject — the  assess¬ 
ment  of  M&S  investment.  Understand¬ 
ably,  each  sector  had  a  slightly  different 
motivation  or  incentive  to  invest  in  M&S. 
One  company  was  concerned  about  long¬ 
term  applicability  and  the  transference  of 
technology  to  future  programs.  Another 
was  in  the  business  of  capturing  data  and 
shaping  models  and  the  requisite  simula¬ 
tion  runs  to  satisfy  client  needs.  They  were 
using  training  models  and  virtual  reality 
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technology  as  a  means  to  reduce  training 
costs  and  time.  Another  believes  a  judi¬ 
cious  application  of  M&S  reduces  direct 
manufacturing  costs.  For  example,  on  the 
JSF  program,  simulations  are  improving 
mechanical  tolerances  such  that  develop¬ 
ers  project  shim  stock  weight  reduction 
from  an  average  of  40  pounds  per  aircraft 
(as  is  the  case  with  the  F— 16)  down  to  less 
than  1  pound.  Such  projections  are  rea¬ 
sonable,  based  on  actual  data  from  Boeing 
777  design  and  projection. 

While  there  are  some  common  appli¬ 
cations  industry-wide,  DoD  is  not  focused 
on  profit,  but  on  performance  and  total 
ownership  costs.2  We  surveyed  a  large 
number  of  DoD  program  managers,  with 
programs  in  various  stages  of  the  acqui¬ 
sition  process.  The  survey  requested  gen¬ 
eral  information  from  the  program  man¬ 
agers  regarding  how  they  made  decisions 
on  M&S  investments.  Additionally,  in  an 
effort  to  maintain  balance,  we  also  sur¬ 
veyed  several  government  contractors.  We 


also  conducted  a  literary  search  to  cap¬ 
ture  the  body  of  knowledge  in  non-govem- 
ment  organizations  related  to  justifying 
M&S  investments. 

Results  of  Surveys 

In  general,  survey  responses  indicated 
that  program  managers  are  investing  in 
M&S  to  support  program  development; 
however,  most  of  the  investment  decisions 
were  based  on  intuition  or  need-based  fac¬ 
tors.  Most  decisions  were  made  without 
detailed  quantitative  analysis.  Ostensibly, 
program  managers  accepted  modest 
investment  in  M&S  because  they  believed 
that  it  would  benefit  the  program.  These 
efforts,  however  commendable,  lacked  a 
methodical  cost-benefit  analysis.  Addi¬ 
tionally,  the  lack  of  a  structured  business 
case  analysis  made  it  difficult,  if  not  im¬ 
possible,  for  program  managers  to  articu¬ 
late  or  substantiate  their  investment  strat¬ 
egy.  In  some  cases  it  came  down  to  a  pro¬ 
gram  manager  wishing  to  explore  M&S 
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without  knowing  what  benefits  might  be 
achieved.  This  is  not  really  taking  a  risk; 
it  is  more  an  exploratory  probe  into  a  new 
field.  These  program  managers  were  em¬ 
bracing  new  technology — but  without  a 
sound  business  case  approach,  they  could 
not  really  assess  if  the  investment  would 
realize  considerable  benefits  or  generate 
prohibitive  costs. 

Most  program  managers  justified  their 
M&S  investment  based  on  one  or  more 
of  the  following: 

•  reducing  design  cycle  time; 

•  augmenting  or  replacing  physical  tests; 

•  helping  resolve  limitations  of  funds, 

assets  or  schedules;  or 

•  providing  insight  into  issues  that  were 

impossible  or  impracticable  to  examine 

in  other  ways. 

Much  of  the  feedback  reflected  that 
program  managers  had  tried  to  examine 
costs  and  measure  them  against  bene¬ 
fits — but  without  the  help  of  any  business 
case  analysis  format.  Consequently,  ap¬ 
proaches  and  results  varied.  Few  reported 
to  have  used  a  disciplined  approach.  The 
use  of  inconsistent  applications  or  ap¬ 
proaches  leads  to  mixed  results  that  cannot 
be  readily  compared  or  evaluated.  The 
majority  of  respondents  suggested  the 
question  of  using  M&S  was  not  one  of 
“Should  I”  but  rather,  “Flow  can  I?”  This 
demonstrates  general  acceptance  that 
M&S  is  required  in  an  efficiently  man¬ 
aged  program.  But  it  begs  the  question  “at 
what  cost?”  Indeed,  the  cost  of  investment 
may  be  prohibitively  expensive,  or  there 
may  be  only  the  most  marginal  long-term 


benefits  to  the  department.  In  such  cases, 
the  decision  maker  should  select  from  al¬ 
ternatives,  be  it  foregoing  M&S,  partner¬ 
ship  with  other  program  managers  to  share 
costs,  or  leveraging  the  investment  of  oth¬ 
ers.  This  is  not  a  moot  point,  given  DoD’s 
fiscal  limitations. 

Program  managers  want  confirmation 
that  investment  in  M&S  will  yield  direct 
savings  within  their  budgets.  Considering 
that  much  of  the  benefit  of  M&S  invest¬ 
ment  is  intangible,  traditional  measure¬ 
ment  approaches  may  not  provide  an 
accurate  assess¬ 
ment.  Addition¬ 
ally,  program 
managers  may 
be  understand¬ 
ably  too  paro¬ 
chial  in  that,  if 
the  return  is  not 
significant  for 
their  immediate 
project  needs,  they  may  dismiss  these 
tools.  But  projects  are  relatively  short¬ 
lived — we  may  very  well  be  missing  some 
of  the  longer-term  residual  benefits.  While 
traditionally  the  benefits  of  M&S  tend  to 
be  discussed  in  terms  of  return  on  invest¬ 
ment,  several  alternatives  for  business  case 
analysis  can  just  as  effectively  justify 
M&S  investments.  The  challenge  is  to 
define  an  appropriate  strategy  and 
priorities  to  address  the  business  value 
proposition. 

A  disciplined  approach  and  methodol¬ 
ogy  has  many  benefits.  It  can  help  bring 
the  aggregate  benefits  into  focus  and 
strengthen  the  argument  for  M&S  invest¬ 
ment.  A  business  case  analysis  provides  a 
convenient  mechanism  for  project  man¬ 
agement.  It  can  be  an  easy-to-follow,  logi¬ 
cal  thread.  It  also  lays  the  groundwork  for 


"  Program 
managers  want 
confirmation  that 
investment  in  M&S 
will  yield  direct 
savings  within 
their  budgets." 
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others  to  attain  information  that  will  help 
their  respective  programs.  This  approach 
forces  a  timeline,  captures  benefits,  and 
enables  authorities  to  decide  if  the  return 
is  worth  the  pursuit. 

Making  the  Business  Case 


Corporate  America  is  taking  a  methodi¬ 
cal  approach  to  investment  decisions 
regarding  M&S.  Although  they  are  prima¬ 
rily  profit  driven,  they  share  common 
maxims  of  production — such  as  cost 
reduction,  efficiency,  and  cost  avoid¬ 
ance — to  that  of  DoD.  Essentially,  a  busi¬ 
ness  case  will  assist  the  program  manager 
in  evaluating  which  of  a  number  of  logi¬ 
cal  packages  of  alternatives  will  best  meet 
the  program’s  objective. 


procedures  for  economic  analysis.  Some 
services  may  also  have  supplemental  guid¬ 
ance  (U.S.  Army  cost  and  Economic 
Analysis  Center,  1995). 

A  Busin  ess  Case  Framework 


Given  our  research,  the  results  of  sur¬ 
veys,  discussions  with  industry  and  our 
literature  search,  we  recommend  a  seven- 
step  process  for  assessing  the  utility  of 
M&S  investments: 

•  Establish  a  baseline. 

•  Establish  a  vision  and  direction. 

•  Quantify  the  costs  and  benefits  of 
alternatives/capabilities. 


Existing  Guidelines  and  Instructions 

The  survey  revealed  that  although  DoD 
has  issued  guidance  on  this  subject,  few 

program  man¬ 
agers  reported 
that  they  were 
following  it. 
DoD  guidance 
on  investment 
guidance  is 
provided  by 
DoD  Instruc¬ 
tion  (DoDI) 
Number  7041.3,  Economic  Analysis  for 
Decisionmaking,  (1995).  Enclosure  3  of 
the  DoDI,  Procedures  for  Economic 
Analysis ,  provides  an  insightful  overview 
of  methodology,  criteria  and  a  discussion 
of  sensitivity  analysis  (1995).  The  Gen¬ 
eral  Services  Administration’s  Informa¬ 
tion  Technology  Capital  Planning  and 
Investment  Guide,  (1998),  also  provides 


•  Evaluate  alternatives. 

•  Conduct  sensitivity  analysis. 

•  Develop  a  migration  strategy. 

•  Monitor  the  process  and  continue  to 
assess  results  through  formalized 
feedback. 

Step  1.  Establishing  a  Baseline 

As  with  many  other  decision-making 
processes,  the  first  step  is  to  establish  an 
accurate  baseline.  The  baseline  provides 
a  benchmark  from  which  decisions  will 
be  weighed  and  assessed.  The  baseline 
should  include  a  clear  enunciation  of 
assumptions  and  constraints.  Assumptions 
are  explicit  statements  describing  the 
present  and  future  environment.  They  re¬ 
duce  complex  situations  into  manageable 
proportions.  These  assumptions  normally 


"The  baseline 
provides  a  bench¬ 
mark  from  which 
decisions  will  be 
weighed  and 
assessed." 
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provide  some  comment  on  the  estimated 
future  workload,  the  useful  life  of  the  in¬ 
vestment  or  system,  and  the  period  of  time 
over  which  alternatives  will  be  compared. 
Assumptions  should  also  discuss  sunk 
costs  and  realized  benefits,  but  are  not 
included  as  part  of  the  baseline.  Con¬ 
straints  are  those  factors  that  limit  alter¬ 
natives.  Normally  they  are  expressed  in 
terms  of  time,  finances,  institutional  or 
regulatory  statutes,  or  directives  and 
physical  plant  and  assets. 

The  baseline  must  identify  the  “higher- 
value”  portions  of  a  program  in  order  to 
evaluate  the  appropriateness  of  various  al¬ 
ternatives.  For  example,  in  the  case  of  an 
aircraft,  75  percent  of  program  hours 
might  be  expended  on  the  air  vehicle  team. 
On  closer  examination,  one  might  find  that 
45  percent  of  that  time  goes  into  airframe 
design,  and  of  that  figure,  90  percent  is 
expended  on  mid-fuselage  development. 
With  this  informed  examination,  the  pro¬ 
gram  manager  will  be  better  able  to  allo¬ 
cate  M&S  spending  where  it  will  offer  the 
greatest  potential  savings  or  benefits.  The 
baseline  must  deteimine  these  high  value 
areas  for  effective  business  case  analysis. 
These  program  specifics  form  the  drivers 
to  the  program,  which  in  turn  drive  the 
investment  process. 

Step  2.  Establishing  Vision  and  Direction 

One  must  look  to  the  future,  then  bridge 
the  gap  between  present  knowledge  and 
that  required  to  make  future  products  a 
reality.  A  program  manager  must  estab¬ 
lish  the  program’s  vision.  As  with  other 
technologies,  the  program  manager’s 
vision  should  consider  how  M&S  tools 
can  improve  program  costs,  scheduling, 
and  performance,  and  whether  scientific 
knowledge  exists  to  support  such  M&S 


investment.  The  vision  should  drive  what 
the  M&S  tools  should  be  trying  to  solve, 
not  the  other  way  around. 

For  example,  although  the  Big  Three 
auto  companies  in  Detroit  are  producing 
very  similar  products,  each  is  using  M&S 
in  very  different  ways,  based  upon  their 
strategic  visions.  One  is  concentrating 
heavily  on  using  M&S  in  design  at  a  single 
location;  another  is  concentrating  on  mov¬ 
ing  large  amounts  of  digital  information 
around  the  world  in  order  to  develop  a  glo¬ 
bal  engineering  capability;  and  the  third 
is  heavily  in¬ 
vesting  in  re¬ 
ducing  the  costs 
of  manufactur¬ 
ing.  Of  course 
each  is  doing 
some  or  all  of 
these,  but  the 
vision  provides  the  focus  for  allocating 
their  scarce  resources.  It  clearly  identifies 
what  is  to  be  achieved,  without  dictating 
how  it  will  be  done. 

Step  3.  Quantify  the  Costs  and  Benefits 
of  Alternatives  and  Capabilities 

Alternatives  are  logical  packages  of 
initiatives  that  work  well  together  (Kid- 
well,  1998).  One  alternative  is  the  status 
quo — that  is,  what  we  identified  as  the 
baseline  in  Step  1 .  In  some  cases  the  base¬ 
line,  often  a  physical  test,  can  be  more  cost 
effective  than  the  use  of  M&S.  Other 
alternatives  should  represent  various  com¬ 
binations  of  M&S  tools  that  help  achieve 
the  vision.  In  determining  alternatives,  one 
must  consider  both  immediate  and  long¬ 
term  effects.  First,  what  is  it  the  program 
needs  to  perform  better?  Second,  what 
does  the  program  need  in  order  to  survive 
until  the  next  stage?  Most  certainly,  a 
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program  must  satisfy  a  requirement,  but 
it  must  also  endure  and  survive  each  step 
of  the  process.  It  does  no  good  for  a  pro¬ 
gram  to  bankrupt  itself  with  massive  un¬ 
focused  M&S  investment  early  on.  Each 
investment  must  produce  value  during  a 
time-frame  that  is  appropriate  for  the  pro¬ 
gram.  Reality  requires  meeting  near-term 
milestones.  The  alternatives  must  also 
consider  the  technological  advances  in 
M&S  tools  that  might  occur.  A  program 
might  not  be  justified  in  spending  huge 
sums  of  money  on  M&S  technology  that 
will  be  superceded  and  rendered  obsolete 
in  2-5  years. 

The  program  manager  must  identify  all 
costs  that  are  incident  to  achieving  each 
alternative.  Models  and  simulations  can 
be  expensive  to  develop,  particularly  in 
domains  where  the  scientific  principles  are 
not  fully  understood  as  applied  to  the 
problem.  While  additional  research  can  fill 
the  knowledge  voids,  the  cost  of  this 
research  must  be  factored  into  the  analy¬ 
sis  of  alternatives.  These  should  include 

the  opportu¬ 
nity  costs  of 
assets  and  re¬ 
sources,  which 
are  the  alterna¬ 
tive  value  fore¬ 
gone  when  an 
asset  is  used 
for  other  purposes  (DoDI.  1995).  They 
also  include  nonrecurring  and  recurring 
costs.  Life-cycle  costs  should  include  all 
costs,  nonrecurring  and  recurring,  that 
occur  over  the  life  of  an  alternative  (Gen¬ 
eral  Services  Administration,  1998). 

Identifying  alternatives  may  be  the 
most  difficult  portion  of  the  process.  Ben¬ 
efits  must  be  viewed  primarily  in  terms 
of  measurable  value.  Expected  benefits 


should  flow  from  the  clear  operating  vi¬ 
sion  developed  in  Step  2.  Enigmatically, 
there  are  both  quantifiable  and  unquan- 
tifiable  benefits.  The  former  have  some 
tangible  or  readily  identified  returns;  the 
latter  have  less  so.  Additionally,  there  may 
be  benefits  that  have  no  intrinsic  value  to 
one  program  but  provide  value  to  others. 
We  call  these  external  benefits. 

Quantifiable  benefits.  These  include 
cost  savings,  time  improvement,  accelera¬ 
tion  of  deliverables,  quality  enhancement 
and,  in  most  cases,  cost  avoidance  that  is 
directly  related  to  the  program.  The  alter¬ 
natives  must  also  consider  existing  sys¬ 
tems  and  programs.  If  we  are  to  measure 
improvements  from  an  “as-is  baseline” — 
we  need  not  start  from  ground  zero.  It  may 
be  possible  for  program  managers  to  look 
at  M&S  initiatives  in  other  programs, 
assess  their  applicability,  and  leverage 
them  for  success.  Cost  associated  with 
these  alternatives  should  be  less,  given  that 
a  majority  of  the  investment  would  be  a 
sunk  cost  borne  by  others.  Similarly,  pro¬ 
gram  managers  must  consider  whether 
partnering  with  another  program,  thereby 
sharing  costs,  is  a  possible  alternative  to 
reduce  up-front  investment.  Program  man¬ 
agers  should  ensure  they  have  examined 
all  potential  benefits  by  using  published 
references  and  experts  in  the  field  of  cost 
analysis. 

Unquantifiable  benefits.  Traditionally, 
we  have  considered  the  issues  of  risk 
reduction,  organizational  efficiency,  tech¬ 
nology  transference,  product  safety,  and 
environmental  impact  reductions  as  un¬ 
measurable  and  therefore  unquantifiable. 
However,  these  are  important  issues,  and 
program  managers  must  consider  them  in 
their  analysis.  To  illustrate  this  point,  we 
will  address  technology  transference. 
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The  Grizzly 


M&S  technology  transference  can  sig¬ 
nificantly  influence  costs,  but  in  today’s 
DoD  environment  it  has  yet  to  receive 
adequate  attention.  Given  the  shrinking 
public  purse  and  the  demand  for  greater 
accountability  and  responsibility  for  the 
dispersal  of  funds,  all  program  managers 
must  show  due  diligence  in  their  public 
spending.  They  must  consider  the  residual 
benefits  of  technology  transference. 

Some  M&S  investment  might  be  of  use 
to  other  projects  and  program  managers. 
For  example,  the  Grizzly3  program  man¬ 
ager  invested  heavily  in  chassis  M&S  to 
support  short-term  design  and  perfor¬ 
mance  analysis.  This  M&S  investment  re¬ 
sulted  in  $21  million  worth  of  quantifi¬ 
able  benefits  to  the  Grizzly  program.  The 
Grizzly  program  manager  funded  the 
M&S  effort  through  internal  reallocation 
of  funds.  The  program  manager’s  supervi¬ 
sor,  program  manager  Combat  Mobility 


Systems,  recognized  the  potential  for  the 
use  of  these  models  for  both  other  pro¬ 
gram  requirements  and  in  other  programs 
sharing  the  Grizzly’s  chassis.  Program 
manager  Combat  Mobility  Systems  lever¬ 
aged  the  Grizzly  Program’s  M&S  invest¬ 
ment,  securing  funding  to  expand  the  ap¬ 
plicability  of  the  initial  investment  into 
other  programs  and  to  support  other  long¬ 
term  Grizzly  requirements. 

When  forecasting  near-term  savings  in 
design  and  production  costs,  one  of  our 
surveyed  companies  accrued  substantial 
unquantifiable  benefits.  The  engineers 
made  a  substantial  leap  in  M&S  knowl¬ 
edge  when  learning  how  to  define  data 
needs,  how  to  shape  models,  and  how  to 
refine  simulation  runs,  to  narrow  the  band¬ 
width  of  problem  solving.  The  resulting 
expertise,  data,  and  process  could  be  ap¬ 
plied  to  future  projects.  Not  surprisingly, 
the  company’s  models  and  data  bank  are 
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the  envy  of  the  industry.  This  was  an 
unquantifiable  gain.  Another  unquanti- 
fiable — or  at  least  indiscrete — benefit  of 
M&S  is  the  competitive  advantage  it  pro¬ 
vides.  This  relates  back  to  establishing  a 
clear  vision  so  that  early  investments  will 
lead  the  company  to  where  it  wants  to  be 
in  the  future. 

Unfortunately,  many  program  man¬ 
agers  dismiss  the  concept  of  unquanti¬ 
fiable  benefits,  program  managers  rarely 
track  these  benefits,  or  those  outside  the 
program’s  realm  with  any  real  vigor.  They 
don’t  afford  them  reasonable  weight  when 
analyzing  costs  and  their  alternatives.  Simi¬ 
larly,  external  benefits  may  exist,  not  only 
to  service  DoD  at  large,  but  to  external 
agencies  and  businesses. 

External  benefits.  These  arc  benefits 
which  do  not  bring  direct  return  or  sav¬ 
ings  to  the  unique  program  being  man¬ 
aged,  but  have  applicability  beyond  the 

program  man¬ 
ager’s  purse. 
As  mentioned 
previously, 
many  M&S  in¬ 
itiatives  and 
their  products 
can  either  be 
modified  or  di¬ 
rectly  transferred  to  other  programs. 
Again,  looking  at  the  Grizzly  program,  the 
contractor  supporting  the  program  man¬ 
ager  (United  Defense  Limited  Partner¬ 
ship)  developed  a  common  product  model 
database  that  benefited  efforts  at  Aberdeen 
Proving  Ground,  The  Army  Warfighting 
Analysis  and  Integration  Center,  Water¬ 
ways  Experiment  Station  and  National 
Training  Center  projects.  Thus  there  is  a 
residual  savings  for  follow-on  users.  The 
surveyed  program  managers  did  identify 


a  problem  with  the  high  cost  of  collecting 
data  and  maintaining  the  database.  While 
the  cost  of  performing  this  might  be  high, 
or  perhaps  even  prohibitive  to  one  project, 
it  could  be  cost  effective  to  several  other 
end  users.  We  must  overcome  the  institu¬ 
tional  bias  that  forces  program  managers 
to  ignore  external  benefits.  A  program 
manager  has  no  incentive  to  take  on  an 
M&S  investment  unless  he  or  she  can  jus¬ 
tify  the  expense  from  the  existing  (and 
often  cash-strapped)  program.  The  follow¬ 
ing  example  demonstrates  how  external 
benefits  can  show  marked  savings  to  DoD 
and  the  public. 

For  years  the  Aberdeen  Test  Center 
(ATC)  put  vehicles  through  multiple  runs 
over  ground  to  determine  wear  and  tear 
on  parts  and  the  resultant  performance 
degradation.  This  required  a  large  num¬ 
ber  of  personnel,  vehicles  and  time  to  log 
thousands  of  miles  to  achieve  statistically 
significant  results.  Since  then,  the  ATC 
completed  an  intensive  project  where  data 
was  collected  describing  the  complete  pro¬ 
file  of  the  course.  Subsequently,  engineers 
built  the  models  and  now  conduct  or  aug¬ 
ment  many  of  these  tests  on  a  virtual  prov¬ 
ing  ground  using  simulations  in  lieu  of 
hardware.  The  simulations  are  so  accurate 
that  they  have  been  able  to  document  mil¬ 
lions  of  dollars  in  cost  avoidance  for  test¬ 
ing  of  Army  programs,  while  concomi¬ 
tantly  helping  the  Army  make  the  requi¬ 
site  decisions  for  product  and  performance 
improvements.  Not  surpri singly,  others 
outside  the  DoD,  including  private  indus¬ 
try,  insurance  coiporations,  and  the  De¬ 
partment  of  Transportation,  also  want  to 
use  this  product.  (According  to  DoDI 
7041.3,  societal  costs  and  benefits  outside 
the  federal  government  are  usually  not 
included  in  a  DoD  analysis). 
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Although  this  program  has  universal 
application,  ATC  had  to  offset  the  costs 
of  this  M&S  project  through  internal  sav¬ 
ings  in  manpower  and  overhead,  rather 
than  being  permitted  to  share  the  cost  with 
other  programs.  An  alternative  strategy 
would  be  to  identify  potential  users  in 
advance  and  share  the  developmental 
costs.  Undeniably,  program  managers 
should  take  the  first  step  to  accrue  direct 
benefits  to  their  programs.  But  they  con¬ 
tinue  to  bypass  transferable  benefits  sim¬ 
ply  because  direct  program  constraints 
preclude  further  investment  of  resources. 

Perhaps  the  real  value  of  identifying 
quantifiable  and  unquantifiable  benefits  is 
in  helping  others  outside  the  program  to 
realize  potential  synergies  of  reuse.  For 
example,  the  program  executive  officer, 
who  is  charged  with  program  oversight, 
will  have  better  visibility  into  require¬ 
ments  and  the  potential  benefits.  Fie  or  she 
can  more  accurately  assess  M&S  invest¬ 
ment  in  relation  to  a  broader  sphere  of 
programs.  Operational  analysis  and  train¬ 
ing  are  just  a  few  examples.  Many  of  these 
benefits,  while  external  to  an  individual 
program  manager,  may  be  internal  benefit 
from  the  program  executive  officer’s  per¬ 
spective.  Armed  with  this  information,  a 
program  executive  officer  may  choose  to 
redirect  funding  from  other  sources  into 
the  program,  and/or  direct  a  program  man¬ 
ager  to  take  a  course  of  action  which  may 
not  be  cost-effective  in  a  micro  perspec¬ 
tive;  but  will  bring  an  aggregate  gain  that 
far  outweighs  the  individual  investment. 
But  sound  management  is  predicated  on 
program  managers  providing  the  program 
executive  officer  with  data  and  informa¬ 
tion  drawn  from  the  program  manager’s 
business  case  analysis. 


Step  4.  Evaluate  Alternatives 

We  must  compare  the  costs  and  ben¬ 
efits  of  each  alternative  and  rank  them. 
Such  comparisons  must  be  accomplished 
using  both  quantitative  and  qualitative 
techniques  and  criteria.  Quantitative  tech¬ 
niques  include  net  present  value,  benefit 
cost  ratio,  return  on  investment,  payback 
method,  internal  rate  of  return,  hurdle  rate, 
and  cost  effectiveness  analysis.4  Qualita¬ 
tive  evaluation 
considerations 
such  as  relation¬ 
ship  to  business 
strategy,  sched¬ 
ule  risk,  organi¬ 
zational  and 
technical  risks, 
social  benefits, 
and  legal  and  regulatory  requirements  may 
greatly  alter  the  quantitative  ranking. 

The  choice  of  appropriate  tools  is 
program  and  situation  dependent,  and  can 
greatly  influence  the  outcome  of  the 
analysis.  These  tools  will  aid  decision 
makers  in  accurately  evaluating  all  alter¬ 
natives  such  that  all  costs  and  benefits  are 
viewed  on  a  level  playing  field.  In  gen¬ 
eral,  each  feasible  alternative,  life-cycle 
costs  and  benefits,  are  adjusted  using 
discount  factors  to  account  for  the  time 
value  of  money.  A  complete  analysis  prop¬ 
erly  relates  quantitative  and  qualitative 
factors.  Given  the  importance  of  these 
choices,  one  should  seek  expert  advice  and 
guidance  before  proceeding. 

Step  5.  Conduct  Sensitivity  Analysis 

Sensitivity  analysis  is  an  essential  step 
in  the  decision  process,  as  it  accounts  for 
ever-present  uncertainties.  Such  analysis  re¬ 
peats  the  above  evaluation  of  alternatives 
with  changes  to  the  uncertain  variables 
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and  examines  the  effect  on  the  final  deci¬ 
sion.  The  outcome  will  provide  a  better 
understanding  of  the  robustness  of  the 
output. 

Sensitivity  analysis  is  highly  recom¬ 
mended,  even  if  there  appeal-  to  be  sig¬ 
nificant  differences  among  the  alterna¬ 
tives,  because  an  apparently  superior  so¬ 
lution  may  be  very  sensitive  to  changes 
in  a  single  variable.  Sensitivity  analysis 
is  required  when  differences  among  alter¬ 
natives  are  less  obvious  and  may  be  to¬ 
tally  driven  by 
variability  of 
key  input  fac¬ 
tors.  The  key 
factors  to  be 
tested  may  in¬ 
clude,  but  are 
not  limited  to, 
project  or  pro¬ 
gram  length,  volume  or  quantity  and  mix 
of  production  units,  requirements,  con¬ 
figurations,  assumptions,  and  discount 
rates  and  other  economic  factors. 

Step  6.  Developing  a  Migration  Strategy 

After  determining  the  best  alternative, 
one  must  develop  a  sound  implementation 
plan  to  migrate  the  “winning  strategy”  into 
the  program.  This  plan  must  incorporate 
a  systematic  approach  whereby  the  devel¬ 
oper  plans  to  implement  the  identified 
drivers  and  capture  the  expected  benefits. 
Implementation  of  the  migration  strategy 
will  undoubtedly  force  changes  to  the 
program’s  plan  and  budget.  If  a  new  tool 
or  process  is  expected  to  save  money,  then 
those  savings  should  be  subtracted  from 
that  part  of  the  program  budget  and 
reassigned  elsewhere  as  an  up-front  action. 


Step  7.  Monitoring  and  Assessment 
through  Formalized  Feedback 

The  final  step  in  developing  a  business 
case  will  be  to  create  metrics  to  assess 
progress  toward  the  overall  vision.  These 
metrics  should  be  tied  to  the  changes  made 
to  the  program’s  acquisition  plan,  to  pro¬ 
vide  timely  feedback  on  their  success  in 
meeting  desired  results  in  performance, 
schedule,  and  cost.  Performance  metrics 
should  stem  from  the  needs  and  require¬ 
ments  that  alternatives  are  fulfilling,  and 
should  address  the  benefits  they  are  ex¬ 
pected  to  provide.  Schedule  and  cost 
metrics  must  also  be  developed  to  help 
ensure  programs  adhere  to  planned  costs 
and  schedules  (Kidwell,  1998;  DoD, 
1995;  GSA,  1998). 

Program  managers  face  tremendous 
pressure  to  bring  a  product  into  use.  While 
it  is  the  program  manager  who  can  best 
provide  monitoring  input,  funding  limits 
and  timelines  debilitate  an  aggregate  ap¬ 
proach  to  monitoring.  Unquestionably, 
program  managers  should  consider  the 
entire  life  cycle  of  the  project.  But  in  the 
present  acquisition  environment  there  is 
little  incentive  to  do  this.  Program  man¬ 
agers  are  the  lynchpin  to  success,  since 
they  hold  all  of  the  program-specific 
information. 

Program  managers  must  deliver  their 
programs  with  complementary  benefits 
first.  This  is  their  true  priority,  but  they 
should  also  identify  real  or  potential 
external  benefits  up  the  management  chain 
to  the  program  executive  officer.  That 
office  can  then  make  more  informed 
decisions  on  the  macro  benefits.  Program 
managers  should  consider  increasing  in¬ 
vestment  earlier  in  the  program  if  the  busi¬ 
ness  case  strongly  indicates  downstream 
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savings  as  a  result.  Program  executive 
officers  can  provide  the  attendant  over¬ 
sight  and  direction,  with  a  requisite  real- 
location  of  funds  when  it  is  in  DoD’s  best 
interest  to  do  so.  DoD  has  made  some  mar¬ 
ginal  progress,  as  historically,  program 
managers  did  not  worry  about  maintain¬ 
ability  and  sustainability  issues.  Now 
Gansler’s  revolution  in  military  affairs  is 
mandated  through  specifications  and  ex¬ 
pectations  in  our  contracts.  Acquisition 
decision  makers  must  shift  expectations 
as  M&S  technology  enables  more  in¬ 
formed  tradeoff  decisions  against  such 
things  as  disposal  costs.  We  must  include 
this  approach  in  our  business  case  analy¬ 
sis.  This  requires  a  fundamental  attitudi- 
nal  change  not  only  on  the  part  of  pro¬ 
gram  managers,  but  also  for  the  entire 
acquisition  team  and  DoD. 

Monitoring  needs  to  be  a  truly  inte¬ 
grated  process,  with  all  elements  actively 
involved.  A  sharing  of  analysis,  combined 
with  a  DoD  commitment  to  maximize  and 
optimize  any  potential  benefits  of  M&S 
technology,  will  bring  unprecedented 
reward — in  cheaper,  better,  stronger  prod¬ 
ucts  and  the  associated  prudence  in 
managing  the  public  purse. 

Conclusions 


Often,  finding  the  solution  to  complex 
technological  problems  is  a  game  of 
chance,  where  there  are  a  limited  number 
of  variables  but  a  near  limitless  combina¬ 
tion  of  these  to  bring  about  technological 
breakthroughs.  M&S  permits  the  program 
manager  to  experiment  with  a  larger  num¬ 
ber  of  possibilities  without  undue  risk. 
Once  the  essential  data  is  determined,  col¬ 
lected,  and  then  shaped  within  a  model, 


the  simulations  provide  a  tremendous 
advantage  over  traditional  methods  of  trial 
and  error. 

The  use  of  M&S  in  program  manage¬ 
ment  is  no  longer  reserved  for  programs 
on  the  cutting  edge  of  technological 
development,  nor  is  it  simply  an  “experi¬ 
mental  tool.”  Both  industry  and  the  gov¬ 
ernment  should  rely  on  sound  business 
practices  for  success.  Both  should  also 
realize  that  M&S  is  an  established  busi¬ 
ness  tool,  and  that  M&S  investment  justi¬ 
fication  should  be  based  on  a  reasoned 
cost-benefit  analysis. 

Unfortunately  establishing  the  cost- 
benefit  relation¬ 
ship  of  M&S  in¬ 
vestments  can 
be  just  as  daunt¬ 
ing  as  the  man¬ 
agement  of  a 
program  itself. 

The  use  of  a 
business  case 
development 
process  to  jus¬ 
tify  M&S  de¬ 
velopment  pro¬ 
vides  a  flexible 
yet  structured  methodology  for  program 
managers  to  weigh  alternatives.  Business 
case  analysis  permits  the  program  man¬ 
ager  to  justify  investment  decisions  based 
on  traditional  discounted  cash  flow  analy¬ 
ses,  as  a  function  of  externally  imposed 
constraints,  and  risk  reduction.  It  allows 
program  managers  to  capture  not  only 
those  costs  and  benefits  that  are  internally 
quantifiable  and  unquantifiable,  but  to 
address  potential  benefits  that  may  exist 
external  to  the  program.  A  disciplined  ap¬ 
proach  to  making  investment  decisions 
also  provides  a  mechanism  for  those 
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outside  the  program  management  office 
to  examine  and  assess  investments  for 
broader  applicability. 

Our  research  indicates  that  in  a  large 
section  of  the  acquisition  community, 
insufficient  rigor  is  applied  to  M&S 
investment  justification.  We  uncovered  a 
variety  of  tools  and  references  available 
to  program  managers  for  conducting  busi¬ 
ness  case  analyses.  Building  a  business 
case  not  only  helps  program  managers 
ensure  M&S  investment  is  warranted,  but 
serves  as  a  reference  for  others  when  fry¬ 
ing  to  make  similar  investments  or  as  a 
baseline  document  that  other  program 
managers  and  offices  can  use  when  build¬ 
ing  investment  strategies.  Business  case 
analyses  also  build  strong  justifications  to 
defend  M&S  investments  and  bring  rigor 
and  discipline  to  the  program  or  project 
management  processes.  The  seven-step 
procedure  identified  in  the  body  of  this 
article  captures  the  essence  of  available 
guidance,  knowledge,  and  experience  and 
should  provide  program  managers  with  a 
starting  point  when  considering  M&S 
investments. 

Recommendations 


Having  a  clear  understanding  of  the 
state  of  development  we  believe  the  fol¬ 
lowing  recommendations  will  serve  to 
assist  program  managers  with  the  devel¬ 
opment  of  modeling  and  simulation 
investment  strategies  based  on  a  sound 
business  case  development  processes. 


•  Program  managers  need  to  be  encour¬ 
aged  to  add  discipline  and  structure  to 
their  M&S  justification  process.  Ser¬ 
vice  leadership  must  challenge  pro¬ 
gram  managers  to  use  business  case 
development  methodology  to  support 
M&S  investment  decisions. 

•  Program  managers  require  ready  ac¬ 
cess  to  policy  and  guidelines  from  the 
General  Services  Administration,  the 
Office  of  the  Secretary  of  Defense,  and 
each  service  in  order  to  develop  suc¬ 
cessfully  a  business  case  justification. 
We  recommend  incorporation  of  docu¬ 
ments  referenced  in  this  paper  into  the 
Defense  Acquisition  Deskbook  along 
with  a  section  to  serve  as  a  primer  for 
business  case  development. 

•  Program  managers  and  their  staff  need 
adequate  training  in  order  to  properly 
implement  business  case-based  M&S 
investment  strategy  justification.  Ac¬ 
quisition  curriculum  at  service  schools 
and  the  Defense  Systems  Management 
College  should  include  business  case 
development  familiarization  classes. 

•  Service  leadership  should  capture  suc¬ 
cess  stories  and  publish  them  in  appro¬ 
priate  service  and  DoD  journals,  maga¬ 
zines,  other  publications,  and  related 
acquisition  Internet  sites. 
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Endnotes 


1.  Quoted  by  Permission,  Brigadeer 
General  Michael  Hough,  USMC, 
deputy  program  manager  Joint  Strike 
Fighter,  during  an  address  to  the  ICAF 
Acquisition  Class,  March  19,  1999. 

2.  DoD  Total  Ownership  Cost  (TOC)  is 
the  sum  of  all  financial  resources 
necessary  to  organize,  equip,  train, 
sustain,  and  operate  military  forces 
sufficient  to  meet  national  goals  in 
compliance  with  all  laws,  all  policies 
applicable  to  DoD,  all  stnadards  in 
effect  for  readiness,  safety,  and  qual¬ 
ity  of  life,  and  all  other  official  mea¬ 
sures  of  performance  for  DoD  and  it 
components. 


3.  The  Grizzly  Program  is  a  U.S.  Army 
program.  The  Grizzly  is  a  complex 
obstacle  breaching  system  based  on 
the  MI  Abrams  tank  chassis.  It  is  de¬ 
signed  to  support  combined  arms 
maneuver  operations. 

4.  For  a  short  description  of  each,  see 
GSA’s  Information  Technology  Capi¬ 
tal  Planning  and  Investment  Guide. 
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